Migration & Patching Guide

Step-by-step procedure for VM migration during patching

phy17130 Cold Migration

KEPSVPCESB4 - 10.137.171.30 (Cold Migration)

virsh list
virsh shutdown primary-srv1-ceph-mon1
virsh shutdown dr-registry
virsh destroy primary-srv1-ceph-mon1
virsh shutdown primary-srv1-ceph-mon1

AFTER PATCH
virsh list
virsh start kepsvvcmsblbr1
virsh start kepsvvcswmt1
virsh list

NB: Do not start dr-registry & primary-srv1-ceph-mon1 on another node as these are not storage bound.

phy17005 Migration

KEPSVPCLDAP1 -- 10.137.170.5

virsh list
free -h
virsh list --all



virsh --connect qemu+ssh://apizone@KEDSVPCLDAP1:7779/system list

virsh migrate --live keusvvdev01 qemu+ssh://apizone@KEDSVPCLDAP1:7779/system

virsh migrate --live KEUSVVCFCLBR qemu+ssh://apizone@KEDSVPCLDAP1:7779/system

virsh migrate --live KEPSVVCFCRTR1 qemu+ssh://apizone@KEDSVPCLDAP1:7779/system

virsh migrate --live KEPSVVCFCLDP1 qemu+ssh://apizone@KEDSVPCLDAP1:7779/system

virsh migrate --live KEPSVVCFCLBR1 qemu+ssh://apizone@KEDSVPCLDAP1:7779/system

virsh migrate --live KEPSVVCFCNTP1 qemu+ssh://apizone@KEDSVPCLDAP1:7779/system


after patch migrate back from 170105 to 1705


virsh --connect qemu+ssh://apizone@KEPSVPCLDAP1:7779/system list

virsh migrate --live keusvvdev01 qemu+ssh://apizone@KEPSVPCLDAP1:7779/system

virsh migrate --live KEUSVVCFCLBR qemu+ssh://apizone@KEPSVPCLDAP1:7779/system

virsh migrate --live KEPSVVCFCRTR1 qemu+ssh://apizone@KEPSVPCLDAP1:7779/system

virsh migrate --live KEPSVVCFCLDP1 qemu+ssh://apizone@KEPSVPCLDAP1:7779/system

virsh migrate --live KEPSVVCFCLBR1 qemu+ssh://apizone@KEPSVPCLDAP1:7779/system

virsh migrate --live KEPSVVCFCNTP1 qemu+ssh://apizone@KEPSVPCLDAP1:7779/system

phy17160 Migration

phy17160
virsh --connect qemu+ssh://apizone@KEPSVPCESBL2:7779/system list 
free -h

phy17170 

free -h

phy17160

virsh migrate --live voyager001 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvcmsblbr2 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvcswmt2 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvcswno1 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvcswlb1 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvcswnfs qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live mst001 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live mst002 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live nod001 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live nod002 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live apm001 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvckub1 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvckub2 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live KEPSVVCESB3-CEPH qemu+ssh://apizone@KEPSVPCESBL2:7779/system

virsh shutdown primary-srv2-ceph-mon2





AFTER PATCH
phy17170
virsh migrate --live voyager001 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvcmsblbr2 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvcswmt2 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvcswno1 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvcswlb1 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvcswnfs qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live mst001 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live mst002 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live nod001 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live nod002 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live apm001 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvckub1 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvckub2 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live KEPSVVCESB3-CEPH qemu+ssh://apizone@KEPSVPCESBL1:7779/system


ceph -s
ceph df

phy170015 Migration

KEPSVPCLDAP2 - 10.137.170.15

phy17015

free -h
virsh list --all



virsh --connect qemu+ssh://apizone@KEDSVPCLDAP1:7779/system list

virsh list 
virsh shutdown KEPSVVCFCWIN16
virsh migrate --live KEPSVVCFCPRX qemu+ssh://apizone@KEDSVPCLDAP1:7779/system

phy170105

virsh --connect qemu+ssh://apizone@KEPSVPCLDAP2:7779/system list

virsh migrate --live KEPSVVCFCPRX qemu+ssh://apizone@KEPSVPCLDAP2:7779/system

phy17170 Migration

Commands
phy17170

virsh --connect qemu+ssh://apizone@KEPSVPCESBL1:7779/system list 
date
virsh shutdown tzpsvvcvoy01
virsh shutdown kepsvvcswmt3
virsh shutdown mst003
virsh migrate --live kepsvvcswno2 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvcswreg qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live nod003 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvcansible qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live KEUSVVCESB1 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live qpid001 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live nfs001 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live apm002 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh migrate --live kepsvvcmsblbr3 qemu+ssh://apizone@KEPSVPCESBL1:7779/system
virsh shutdown primary-srv3-ceph-mon3

phy17160

virsh shutdown voyager001
rbd mirror image demote libvirt-integ-pool/voyager001
rbd mirror image status libvirt-integ-pool/voyager001


phy17140
rbd mirror image promote libvirt-integ-pool/voyager001
virsh start voyager001


After Patching 
phy17170

virsh migrate --live kepsvvcswno2 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvcswreg qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live nod003 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvcansible qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live KEUSVVCESB1 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live qpid001 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live nfs001 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live apm002 qemu+ssh://apizone@KEPSVPCESBL2:7779/system
virsh migrate --live kepsvvcmsblbr3 qemu+ssh://apizone@KEPSVPCESBL2:7779/system


phy17140

virsh shutdown voyager001
rbd mirror image demote libvirt-integ-pool/voyager001
rbd mirror image status libvirt-integ-pool/voyager001

phy17160
rbd mirror image promote libvirt-integ-pool/voyager001
virsh start voyager001

phy170105 Migration

KEDSVPCLDAP1 - 10.137.170.105

virsh list
Currently no server are running

phy17140 Migration

As no VMs are running, check Ceph health and storage from Slack after patching.

phy17180 Cold Migration

Shutdown Application VMs
virsh shutdown keusvvckubreg
virsh shutdown ugusvvcvoy01
virsh shutdown tzusvvcvoy01
virsh shutdown qpid-uat-vm
virsh shutdown keusvvckubno2
virsh shutdown keusvvckubno1
virsh shutdown keusvvckubmt1
virsh shutdown keusvvckubnfs
Shutdown Ceph VMs
virsh shutdown secondary-srv1-ceph-mon2
virsh shutdown secondary-srv1-ceph-mon1
virsh shutdown secondary-srv1-ceph-mon3


AFTER PATCH 
virsh start secondary-srv1-ceph-mon3
virsh start secondary-srv1-ceph-mon1
virsh start secondary-srv1-ceph-mon2
virsh start keusvvckubnfs

Note

  • Wait 1 minute and SSH to cont160150.
  • If SSH fails, wait another minute and retry.
  • Do not start other VMs until SSH to cont160150 succeeds.

Start Application VMs (order: first → last)

virsh start keusvvckubmt1
virsh start keusvvckubno1
virsh start keusvvckubno2
virsh start qpid-uat-vm
virsh start tzusvvcvoy01
virsh start ugusvvcvoy01
virsh start keusvvckubreg

Post Validation

  1. Check Ceph cluster health → ceph -s
  2. Verify VMs state → virsh list --all
  3. Verify pods on cont160150 → Ensure all UAT pods are running.

phy170115 Migration

KEDSVPCLDAP2 - 10.137.170.115

virsh list
Currently no server are running